Class DfsReader
- java.lang.Object
-
- org.eclipse.jgit.lib.ObjectReader
-
- org.eclipse.jgit.internal.storage.dfs.DfsReader
-
- All Implemented Interfaces:
java.lang.AutoCloseable
,ObjectReuseAsIs
public class DfsReader extends ObjectReader implements ObjectReuseAsIs
Reader to access repository content through.See the base
ObjectReader
documentation for details. Notably, a reader is not thread safe.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
DfsReader.FoundObject<T extends ObjectId>
-
Nested classes/interfaces inherited from class org.eclipse.jgit.lib.ObjectReader
ObjectReader.Filter
-
-
Field Summary
Fields Modifier and Type Field Description private boolean
avoidUnreachable
private DeltaBaseCache
baseCache
private DfsBlock
block
(package private) DfsObjDatabase
db
Database this reader loads objects from.private static java.util.Comparator<DfsReader.FoundObject<?>>
FOUND_OBJECT_SORT
private java.util.zip.Inflater
inf
private DfsPackFile
last
private static int
MAX_RESOLVE_MATCHES
private static java.util.Comparator<DfsObjectToPack>
OFFSET_SORT
private static java.util.Comparator<DfsPackFile>
PACK_SORT_FOR_REUSE
(package private) DfsReaderIoStats.Accumulator
stats
(package private) byte[]
tempId
Temporary buffer large enough for at least one raw object id.-
Fields inherited from class org.eclipse.jgit.lib.ObjectReader
OBJ_ANY, streamFileThreshold
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
DfsReader(DfsObjDatabase db)
Initialize a new DfsReader
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private static boolean
checkGarbagePacks(java.lang.Iterable<ObjectToPack> objects)
private static ObjectLoader
checkType(ObjectLoader ldr, AnyObjectId id, int typeHint)
void
close()
(package private) int
copy(BlockBasedFile file, long position, byte[] dstbuf, int dstoff, int cnt)
Copy bytes from the window to a caller supplied buffer.void
copyObjectAsIs(PackOutputStream out, ObjectToPack otp, boolean validate)
Output a previously selected representation.void
copyPackAsIs(PackOutputStream out, CachedPack pack)
Append an entire pack's contents onto the output stream.private <T extends ObjectId>
java.lang.Iterable<DfsReader.FoundObject<T>>findAll(java.lang.Iterable<T> objectIds)
private java.util.List<DfsObjectToPack>
findAllFromPack(DfsPackFile pack, java.lang.Iterable<ObjectToPack> objects, boolean skipFound)
private <T extends ObjectId>
voidfindAllImpl(DfsObjDatabase.PackList packList, java.util.Collection<T> pending, java.util.List<DfsReader.FoundObject<T>> r)
private java.util.List<DfsPackFile>
garbagePacksForSelectRepresentation()
BitmapIndex
getBitmapIndex()
An index that can be used to speed up ObjectWalks.java.util.Collection<CachedPack>
getCachedPacksAndUpdate(BitmapIndex.BitmapBuilder needBitmap)
Obtain the available cached packs that match the bitmap and update the bitmap by removing the items that are in the CachedPack.(package private) DeltaBaseCache
getDeltaBaseCache()
DfsReaderIoStats
getIoStats()
Get IO statistics accumulated by this reader.<T extends ObjectId>
AsyncObjectSizeQueue<T>getObjectSize(java.lang.Iterable<T> objectIds, boolean reportMissing)
Asynchronous object size lookup.long
getObjectSize(AnyObjectId objectId, int typeHint)
Get only the size of an object.private long
getObjectSizeImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId)
(package private) DfsReaderOptions
getOptions()
java.util.Set<ObjectId>
getShallowCommits()
Returns IDs for those commits which should be considered as shallow.boolean
has(AnyObjectId objectId)
Does the requested object exist in this database?private boolean
hasImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId)
(package private) int
inflate(DfsPackFile pack, long position, byte[] dstbuf, boolean headerOnly)
Inflate a region of the pack starting atposition
.(package private) java.util.zip.Inflater
inflater()
DfsObjectToPack
newObjectToPack(AnyObjectId objectId, int type)
Allocate a newPackWriter
state structure for an object.ObjectReader
newReader()
Construct a new reader from the same data.<T extends ObjectId>
AsyncObjectLoaderQueue<T>open(java.lang.Iterable<T> objectIds, boolean reportMissing)
Asynchronous object opening.ObjectLoader
open(AnyObjectId objectId, int typeHint)
Open an object from this database.private ObjectLoader
openImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId)
(package private) void
pin(BlockBasedFile file, long position)
private void
prepareInflater()
(package private) DfsBlock
quickCopy(DfsPackFile p, long pos, long cnt)
java.util.Collection<ObjectId>
resolve(AbbreviatedObjectId id)
Resolve an abbreviated ObjectId to its full form.private void
resolveImpl(DfsObjDatabase.PackList packList, AbbreviatedObjectId id, java.util.HashSet<ObjectId> matches)
void
selectObjectRepresentation(PackWriter packer, ProgressMonitor monitor, java.lang.Iterable<ObjectToPack> objects)
Select the best object representation for a packer.void
setAvoidUnreachableObjects(boolean avoid)
Advise the reader to avoid unreachable objects.private boolean
skipGarbagePack(DfsPackFile pack)
private java.util.List<DfsPackFile>
sortPacksForSelectRepresentation()
private void
trySelectRepresentation(PackWriter packer, ProgressMonitor monitor, java.lang.Iterable<ObjectToPack> objects, java.util.List<DfsPackFile> packs, boolean skipFound)
(package private) void
unpin()
void
writeObjects(PackOutputStream out, java.util.List<ObjectToPack> list)
Write objects to the pack stream in roughly the order given.-
Methods inherited from class org.eclipse.jgit.lib.ObjectReader
abbreviate, abbreviate, createObjectReachabilityChecker, createReachabilityChecker, getCreatedFromInserter, getStreamFileThreshold, has, open, setStreamFileThreshold
-
-
-
-
Field Detail
-
MAX_RESOLVE_MATCHES
private static final int MAX_RESOLVE_MATCHES
- See Also:
- Constant Field Values
-
tempId
final byte[] tempId
Temporary buffer large enough for at least one raw object id.
-
db
final DfsObjDatabase db
Database this reader loads objects from.
-
stats
final DfsReaderIoStats.Accumulator stats
-
inf
private java.util.zip.Inflater inf
-
block
private DfsBlock block
-
baseCache
private DeltaBaseCache baseCache
-
last
private DfsPackFile last
-
avoidUnreachable
private boolean avoidUnreachable
-
FOUND_OBJECT_SORT
private static final java.util.Comparator<DfsReader.FoundObject<?>> FOUND_OBJECT_SORT
-
OFFSET_SORT
private static final java.util.Comparator<DfsObjectToPack> OFFSET_SORT
-
PACK_SORT_FOR_REUSE
private static final java.util.Comparator<DfsPackFile> PACK_SORT_FOR_REUSE
-
-
Constructor Detail
-
DfsReader
protected DfsReader(DfsObjDatabase db)
Initialize a new DfsReader- Parameters:
db
- parent DfsObjDatabase.
-
-
Method Detail
-
getOptions
DfsReaderOptions getOptions()
-
getDeltaBaseCache
DeltaBaseCache getDeltaBaseCache()
-
newReader
public ObjectReader newReader()
Construct a new reader from the same data.Applications can use this method to build a new reader from the same data source, but for an different thread.
- Specified by:
newReader
in classObjectReader
- Returns:
- a brand new reader, using the same data source.
-
setAvoidUnreachableObjects
public void setAvoidUnreachableObjects(boolean avoid)
Advise the reader to avoid unreachable objects.While enabled the reader will skip over anything previously proven to be unreachable. This may be dangerous in the face of concurrent writes.
- Overrides:
setAvoidUnreachableObjects
in classObjectReader
- Parameters:
avoid
- true to avoid unreachable objects.
-
getBitmapIndex
public BitmapIndex getBitmapIndex() throws java.io.IOException
An index that can be used to speed up ObjectWalks.- Overrides:
getBitmapIndex
in classObjectReader
- Returns:
- the index or null if one does not exist.
- Throws:
java.io.IOException
- when the index fails to load
-
getCachedPacksAndUpdate
public java.util.Collection<CachedPack> getCachedPacksAndUpdate(BitmapIndex.BitmapBuilder needBitmap) throws java.io.IOException
Obtain the available cached packs that match the bitmap and update the bitmap by removing the items that are in the CachedPack.A cached pack has known starting points and may be sent entirely as-is, with almost no effort on the sender's part.
- Specified by:
getCachedPacksAndUpdate
in interfaceObjectReuseAsIs
- Parameters:
needBitmap
- the bitmap that contains all of the objects the client wants.- Returns:
- the available cached packs.
- Throws:
java.io.IOException
- the cached packs cannot be listed from the repository. Callers may choose to ignore this and continue as-if there were no cached packs.
-
resolve
public java.util.Collection<ObjectId> resolve(AbbreviatedObjectId id) throws java.io.IOException
Resolve an abbreviated ObjectId to its full form. This method searches for an ObjectId that begins with the abbreviation, and returns at least some matching candidates. If the returned collection is empty, no objects start with this abbreviation. The abbreviation doesn't belong to this repository, or the repository lacks the necessary objects to complete it. If the collection contains exactly one member, the abbreviation is (currently) unique within this database. There is a reasonably high probability that the returned id is what was previously abbreviated. If the collection contains 2 or more members, the abbreviation is not unique. In this case the implementation is only required to return at least 2 candidates to signal the abbreviation has conflicts. User friendly implementations should return as many candidates as reasonably possible, as the caller may be able to disambiguate further based on context. However since databases can be very large (e.g. 10 million objects) returning 625,000 candidates for the abbreviation "0" is simply unreasonable, so implementors should draw the line at around 256 matches.- Specified by:
resolve
in classObjectReader
- Parameters:
id
- abbreviated id to resolve to a complete identity. The abbreviation must have a length of at least 2.- Returns:
- candidates that begin with the abbreviated identity.
- Throws:
java.io.IOException
- the object store cannot be read.
-
resolveImpl
private void resolveImpl(DfsObjDatabase.PackList packList, AbbreviatedObjectId id, java.util.HashSet<ObjectId> matches) throws java.io.IOException
- Throws:
java.io.IOException
-
has
public boolean has(AnyObjectId objectId) throws java.io.IOException
Does the requested object exist in this database?- Overrides:
has
in classObjectReader
- Parameters:
objectId
- identity of the object to test for existence of.- Returns:
- true if the specified object is stored in this database.
- Throws:
java.io.IOException
- the object store cannot be accessed.
-
hasImpl
private boolean hasImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId) throws java.io.IOException
- Throws:
java.io.IOException
-
open
public ObjectLoader open(AnyObjectId objectId, int typeHint) throws MissingObjectException, IncorrectObjectTypeException, java.io.IOException
Open an object from this database.- Specified by:
open
in classObjectReader
- Parameters:
objectId
- identity of the object to open.typeHint
- hint about the type of object being requested, e.g.Constants.OBJ_BLOB
;ObjectReader.OBJ_ANY
if the object type is not known, or does not matter to the caller.- Returns:
- a
ObjectLoader
for accessing the object. - Throws:
MissingObjectException
- the object does not exist.IncorrectObjectTypeException
- typeHint was not OBJ_ANY, and the object's actual type does not match typeHint.java.io.IOException
- the object store cannot be accessed.
-
checkType
private static ObjectLoader checkType(ObjectLoader ldr, AnyObjectId id, int typeHint) throws IncorrectObjectTypeException
- Throws:
IncorrectObjectTypeException
-
openImpl
private ObjectLoader openImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId) throws java.io.IOException
- Throws:
java.io.IOException
-
getShallowCommits
public java.util.Set<ObjectId> getShallowCommits()
Returns IDs for those commits which should be considered as shallow.- Specified by:
getShallowCommits
in classObjectReader
- Returns:
- IDs of shallow commits
-
findAll
private <T extends ObjectId> java.lang.Iterable<DfsReader.FoundObject<T>> findAll(java.lang.Iterable<T> objectIds) throws java.io.IOException
- Throws:
java.io.IOException
-
findAllImpl
private <T extends ObjectId> void findAllImpl(DfsObjDatabase.PackList packList, java.util.Collection<T> pending, java.util.List<DfsReader.FoundObject<T>> r)
-
skipGarbagePack
private boolean skipGarbagePack(DfsPackFile pack)
-
open
public <T extends ObjectId> AsyncObjectLoaderQueue<T> open(java.lang.Iterable<T> objectIds, boolean reportMissing)
Asynchronous object opening.- Overrides:
open
in classObjectReader
- Parameters:
objectIds
- objects to open from the object store. The supplied collection must not be modified until the queue has finished.reportMissing
- if true missing objects are reported by calling failure with a MissingObjectException. This may be more expensive for the implementation to guarantee. If false the implementation may choose to report MissingObjectException, or silently skip over the object with no warning.- Returns:
- queue to read the objects from.
-
getObjectSize
public <T extends ObjectId> AsyncObjectSizeQueue<T> getObjectSize(java.lang.Iterable<T> objectIds, boolean reportMissing)
Asynchronous object size lookup.- Overrides:
getObjectSize
in classObjectReader
- Parameters:
objectIds
- objects to get the size of from the object store. The supplied collection must not be modified until the queue has finished.reportMissing
- if true missing objects are reported by calling failure with a MissingObjectException. This may be more expensive for the implementation to guarantee. If false the implementation may choose to report MissingObjectException, or silently skip over the object with no warning.- Returns:
- queue to read object sizes from.
-
getObjectSize
public long getObjectSize(AnyObjectId objectId, int typeHint) throws MissingObjectException, IncorrectObjectTypeException, java.io.IOException
Get only the size of an object.The default implementation of this method opens an ObjectLoader. Databases are encouraged to override this if a faster access method is available to them.
- Overrides:
getObjectSize
in classObjectReader
- Parameters:
objectId
- identity of the object to open.typeHint
- hint about the type of object being requested, e.g.Constants.OBJ_BLOB
;ObjectReader.OBJ_ANY
if the object type is not known, or does not matter to the caller.- Returns:
- size of object in bytes.
- Throws:
MissingObjectException
- the object does not exist.IncorrectObjectTypeException
- typeHint was not OBJ_ANY, and the object's actual type does not match typeHint.java.io.IOException
- the object store cannot be accessed.
-
getObjectSizeImpl
private long getObjectSizeImpl(DfsObjDatabase.PackList packList, AnyObjectId objectId) throws java.io.IOException
- Throws:
java.io.IOException
-
newObjectToPack
public DfsObjectToPack newObjectToPack(AnyObjectId objectId, int type)
Allocate a newPackWriter
state structure for an object.PackWriter
allocates these objects to keep track of the per-object state, and how to load the objects efficiently into the generated stream. Implementers may subclass this type with additional object state, such as to remember what file and offset contains the object's pack encoded data.- Specified by:
newObjectToPack
in interfaceObjectReuseAsIs
- Parameters:
objectId
- the id of the object that will be packed.type
- the Git type of the object that will be packed.- Returns:
- a new instance for this object.
-
selectObjectRepresentation
public void selectObjectRepresentation(PackWriter packer, ProgressMonitor monitor, java.lang.Iterable<ObjectToPack> objects) throws java.io.IOException, MissingObjectException
Description copied from interface:ObjectReuseAsIs
Select the best object representation for a packer.Implementations should iterate through all available representations of an object, and pass them in turn to the PackWriter though
PackWriter.select(ObjectToPack, StoredObjectRepresentation)
so the writer can select the most suitable representation to reuse into the output stream.If the implementation returns CachedPack from
ObjectReuseAsIs.getCachedPacksAndUpdate(BitmapBuilder)
it must consider the representation of any object that is stored in any of the offered CachedPacks. PackWriter relies on this behavior to prune duplicate objects out of the pack stream when it selects a CachedPack and the object was also reached through the thin-pack enumeration.The implementation may choose to consider multiple objects at once on concurrent threads, but must evaluate all representations of an object within the same thread.
- Specified by:
selectObjectRepresentation
in interfaceObjectReuseAsIs
- Parameters:
packer
- the packer that will write the object in the near future.monitor
- progress monitor, implementation should update the monitor once for each item in the iteration when selection is done.objects
- the objects that are being packed.- Throws:
MissingObjectException
- there is no representation available for the object, as it is no longer in the repository. Packing will abort.java.io.IOException
- the repository cannot be accessed. Packing will abort.
-
trySelectRepresentation
private void trySelectRepresentation(PackWriter packer, ProgressMonitor monitor, java.lang.Iterable<ObjectToPack> objects, java.util.List<DfsPackFile> packs, boolean skipFound) throws java.io.IOException
- Throws:
java.io.IOException
-
sortPacksForSelectRepresentation
private java.util.List<DfsPackFile> sortPacksForSelectRepresentation() throws java.io.IOException
- Throws:
java.io.IOException
-
garbagePacksForSelectRepresentation
private java.util.List<DfsPackFile> garbagePacksForSelectRepresentation() throws java.io.IOException
- Throws:
java.io.IOException
-
checkGarbagePacks
private static boolean checkGarbagePacks(java.lang.Iterable<ObjectToPack> objects)
-
findAllFromPack
private java.util.List<DfsObjectToPack> findAllFromPack(DfsPackFile pack, java.lang.Iterable<ObjectToPack> objects, boolean skipFound) throws java.io.IOException
- Throws:
java.io.IOException
-
copyObjectAsIs
public void copyObjectAsIs(PackOutputStream out, ObjectToPack otp, boolean validate) throws java.io.IOException, StoredObjectRepresentationNotAvailableException
Output a previously selected representation.PackWriter
invokes this method only if a representation previously given to it byselectObjectRepresentation
was chosen for reuse into the output stream. Theotp
argument is an instance created by this reader's ownnewObjectToPack
, and the representation data saved within it also originated from this reader.Implementors must write the object header before copying the raw data to the output stream. The typical implementation is like:
MyToPack mtp = (MyToPack) otp; byte[] raw; if (validate) raw = validate(mtp); // throw SORNAE here, if at all else raw = readFast(mtp); out.writeHeader(mtp, mtp.inflatedSize); out.write(raw);
- Specified by:
copyObjectAsIs
in interfaceObjectReuseAsIs
- Parameters:
out
- stream the object should be written to.otp
- the object's saved representation information.validate
- if true the representation must be validated and not be corrupt before being reused. If false, validation may be skipped as it will be performed elsewhere in the processing pipeline.- Throws:
java.io.IOException
- the stream's write method threw an exception. Packing will abort.StoredObjectRepresentationNotAvailableException
- the previously selected representation is no longer available. If thrown beforeout.writeHeader
the pack writer will try to find another representation, and write that one instead. If throw afterout.writeHeader
, packing will abort.
-
writeObjects
public void writeObjects(PackOutputStream out, java.util.List<ObjectToPack> list) throws java.io.IOException
Write objects to the pack stream in roughly the order given.PackWriter
invokes this method to write out one or more objects, in approximately the order specified by the iteration over the list. A simple implementation of this method would just iterate the list and output each object:for (ObjectToPack obj : list) out.writeObject(obj)
However more sophisticated implementors may try to perform some (small) reordering to access objects that are stored close to each other at roughly the same time. Implementations may choose to write objects out of order, but this may increase pack file size due to using a larger header format to reach a delta base that is later in the stream. It may also reduce data locality for the reader, slowing down data access. InvokingPackOutputStream.writeObject(ObjectToPack)
will causeObjectReuseAsIs.copyObjectAsIs(PackOutputStream, ObjectToPack, boolean)
to be invoked recursively onthis
if the current object is scheduled for reuse.- Specified by:
writeObjects
in interfaceObjectReuseAsIs
- Parameters:
out
- the stream to write each object to.list
- the list of objects to write. Objects should be written in approximately this order. Implementors may resort the list elements in-place during writing if desired.- Throws:
java.io.IOException
- the stream cannot be written to, or one or more required objects cannot be accessed from the object database.
-
copyPackAsIs
public void copyPackAsIs(PackOutputStream out, CachedPack pack) throws java.io.IOException
Append an entire pack's contents onto the output stream.The entire pack, excluding its header and trailing footer is sent.
- Specified by:
copyPackAsIs
in interfaceObjectReuseAsIs
- Parameters:
out
- stream to append the pack onto.pack
- the cached pack to send.- Throws:
java.io.IOException
- the pack cannot be read, or stream did not accept a write.
-
copy
int copy(BlockBasedFile file, long position, byte[] dstbuf, int dstoff, int cnt) throws java.io.IOException
Copy bytes from the window to a caller supplied buffer.- Parameters:
file
- the file the desired window is stored within.position
- position within the file to read from.dstbuf
- destination buffer to copy into.dstoff
- offset withindstbuf
to start copying into.cnt
- number of bytes to copy. This value may exceed the number of bytes remaining in the window starting at offsetpos
.- Returns:
- number of bytes actually copied; this may be less than
cnt
ifcnt
exceeded the number of bytes available. - Throws:
java.io.IOException
- this cursor does not match the provider or id and the proper window could not be acquired through the provider's cache.
-
inflate
int inflate(DfsPackFile pack, long position, byte[] dstbuf, boolean headerOnly) throws java.io.IOException, java.util.zip.DataFormatException
Inflate a region of the pack starting atposition
.- Parameters:
pack
- the file the desired window is stored within.position
- position within the file to read from.dstbuf
- destination buffer the inflater should output decompressed data to. Must be large enough to store the entire stream, unless headerOnly is true.headerOnly
- if true the caller wants onlydstbuf.length
bytes.- Returns:
- number of bytes inflated into
dstbuf
. - Throws:
java.io.IOException
- this cursor does not match the provider or id and the proper window could not be acquired through the provider's cache.java.util.zip.DataFormatException
- the inflater encountered an invalid chunk of data. Data stream corruption is likely.
-
quickCopy
DfsBlock quickCopy(DfsPackFile p, long pos, long cnt) throws java.io.IOException
- Throws:
java.io.IOException
-
inflater
java.util.zip.Inflater inflater()
-
prepareInflater
private void prepareInflater()
-
pin
void pin(BlockBasedFile file, long position) throws java.io.IOException
- Throws:
java.io.IOException
-
unpin
void unpin()
-
getIoStats
public DfsReaderIoStats getIoStats()
Get IO statistics accumulated by this reader.- Returns:
- IO statistics accumulated by this reader.
-
close
public void close()
Release any resources used by this reader.
A reader that has been released can be used again, but may need to be released after the subsequent usage.
Release the current window cursor.
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in classObjectReader
-
-