Python object-oriented database

ZODB, a Python object-oriented database

Latest release Supported Python versions Build status Coverage status Documentation status

ZODB provides an object-oriented database for Python that provides a high-degree of transparency. ZODB runs on Python 2.7 or Python 3.4 and above. It also runs on PyPy.

  • no separate language for database operations

  • very little impact on your code to make objects persistent

  • no database mapper that partially hides the database.

    Using an object-relational mapping is not like using an object-oriented database.

  • almost no seam between code and database.

ZODB is an ACID Transactional database.

To learn more, visit: https://zodb-docs.readthedocs.io

The github repository is: at https://github.com/zopefoundation/zodb

If you're interested in contributing to ZODB itself, see the developer notes.

Comments
  • Simplify MVCC by determining transaction start time using lastTransac…

    Simplify MVCC by determining transaction start time using lastTransac…

    …tion.

    This implements: https://github.com/zopefoundation/ZODB/issues/50

    Rather than watching invalidations, simply use 1 + the storages lastTransaction, which is equivalent to but much simpler than waiting for the first invalidation after a transaction starts.

    More importantly, it means we can always use loadBefore and get away from load. We no longer have to worry about ordering of invalidations and load() results.

    Much thanks to NEO for pointing the way toward this simplification!

    Implementing this initially caused a deadlock, because DB.open() called Connection.open() while holding a database lock and Connection.open() now calls IStotage.lastTransaction(), which acquires a storage lock. (It's not clear that lastTransaction() really needs a storage lock.) Meanwhile, IStotage.tpc_finish() calls a DB function that requires the DB lock while holding the storage lock. Fixing this required moving the call to Connection.open() outside the region where the DB lock was held.

    To debug the problem above, I greatly improved lock-debugging support. Now all of the ZODB code imports Lock, RLock and Condition from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value, then these are wrapped in such a way that debugging information is printed as they are used. This made spotting the nature of the deadlock easier.

    Of course, a change this basic broke lots of tests. Most of the breakage arises from the fact that connections now call lastTransaction on storages at transaction boundaries. Lots of tests didn't clean up databases and connections properly. I fixed many tests, but ultimately gave up and added some extra cleanup code that violated transaction-manager underware (and the underware's privates) to clear transaction synchonizers in test setup and tear-down. I plan to add a transaction manager API for this and to use it in a subsequent PR.

    This tests makes database and connection hygiene a bit more important, especially for tests, because a connection will continue to interact with storages if it isn't properly closed, which can lead to errors if the storage is closed. I chose not to swallow these errors in Connection, choosing rather to clean up tests.

    The thread debugging and test changes make this PR larger than I would have liked. Apologies in advance to the reviewers.

  • recent changes seem to break winbot

    recent changes seem to break winbot

    http://winbot.zope.org/builders/ZODB_dev%20py_270_win32/builds/2116/steps/test/logs/stdio

    (I plan to update builders to the currently supported platforms)

    (also, @jimfulton you need failure emails from winbot?)

  • tests: Add test for open vs invalidation race

    tests: Add test for open vs invalidation race

    Add test that exercises open vs invalidation race condition that, if happen, leads to data corruption. We are seeing such race happening on storage level in ZEO (https://github.com/zopefoundation/ZEO/issues/166), and previously we've seen it also to happen on Connection level (https://github.com/zopefoundation/ZODB/issues/290). By adding this test to be exercised wrt all storages we make sure that all storages stay free from this race.

    And it payed out. Besides catching original problems from https://github.com/zopefoundation/ZODB/issues/290 and https://github.com/zopefoundation/ZEO/issues/166 , this test also discovered a concurrency bug in MVCCMappingStorage:

    Failure in test check_race_open_vs_invalidate (ZODB.tests.testMVCCMappingStorage.MVCCMappingStorageTests)
    Traceback (most recent call last):
      File "/usr/lib/python2.7/unittest/case.py", line 329, in run
        testMethod()
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/BasicStorage.py", line 492, in check_race_open_vs_invalidate
        self.fail(failure[0])
      File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
        raise self.failureException(msg)
    AssertionError: T1: obj1.value (24)  !=  obj2.value (23)
    

    The problem with MVCCMappingStorage was that instance.poll_invalidations was correctly taking main_lock with intention to make sure main data is not mutated during analysis, but instance.tpc_finish and instance.tpc_abort did not taken main lock, which was leading to committed data to be propagating into main storage in non-atomic way.

    This bug was also observable if both obj1 and obj2 in the added test were always loaded from the storage (added obj2._p_invalidate after obj1._p_invalidate).

    -> Fix MVCCMappingStorage by correctly locking main MVCCMappingStorage instance when processing transaction completion.

    /cc @d-maurer, @jamadden, @jmuchemb

  • Kill leftovers of pre-MVCC read conflicts

    Kill leftovers of pre-MVCC read conflicts

    In the early days, before MVCC was introduced, ZODB used to raise ReadConflictError on access to object that was simultaneously changed by another client in concurrent transaction. However, as doc/articles/ZODB-overview.rst says

    Since Zope 2.8 ZODB has implemented **Multi Version Concurrency Control**.
    This means no more ReadConflictErrors, each transaction is guaranteed to be
    able to load any object as it was when the transaction begun.
    

    So today the only way to get a ReadConflictError should be at commit time for an object that was requested to stay unchanged via checkCurrentSerialInTransaction.

    However MVCCAdapterInstance.load(), instead of reporting "no data", was still raising ReadConflictError for a deleted or not-yet-created object. If an object is deleted and later requested to be loaded, it should be "key not found in database", i.e. POSKeyError, not ReadConflictError. Fix it.

    Adjust docstring of ReadConflictError accordingly to explicitly describe that this error can only happen at commit time for objects requested to be current.

    There were also leftover code, comment and test bits in Connection, interfaces, transact, testmvcc and testZODB, that are corrected/removed correspondingly. testZODB actually had ReadConflictTests that was completely deactivated: commit b0f992fd ("Removed the mvcc option..."; 2007) moved read-conflict-on-access related tests out of ZODBTests, but did not activated moved parts at all, because as that commit says when MVCC is always on unconditionally, there is no on-access conflicts:

    Removed the mvcc option.  Everybody wants mvcc and removing us lets us
    simplify the code a little. (We'll be able to simplify more when we
    stop supporting versions.)
    

    Today, if I try to manually activate that ReadConflictTests via

    @@ -637,6 +637,7 @@ def __init__(self, poisonedjar):
     def test_suite():
         return unittest.TestSuite((
             unittest.makeSuite(ZODBTests, 'check'),
    +        unittest.makeSuite(ReadConflictTests, 'check'),
             ))
    
     if __name__ == "__main__":
    

    it fails in dumb way showing that this tests were unmaintained for ages:

    Error in test checkReadConflict (ZODB.tests.testZODB.ReadConflictTests)
    Traceback (most recent call last):
      File "/usr/lib/python2.7/unittest/case.py", line 320, in run
        self.setUp()
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/testZODB.py", line 451, in setUp
        ZODB.tests.utils.TestCase.setUp(self)
    AttributeError: 'module' object has no attribute 'utils'
    

    Since today ZODB always uses MVCC and there is no way to get ReadConflictError on access, those tests should be also gone together with old pre-MVCC way of handling concurrency.

    /cc @jimfulton

  • Race in Connection.open() vs invalidations -> corrupt data

    Race in Connection.open() vs invalidations -> corrupt data

    Hello up there,

    For wendelin.core v2 I need a way to know at which particular database state application-level ZODB connection is viewing the database. I was looking for that in Connection.open / MVCCAdapter code and noticed concurrency bug that leads to wrong data returned by ZODB to application:

    ---- 8< ---- (zopenrace.py)

    #!/usr/bin/env python
    """Program zopenrace.py demonstrates concurrency bug in ZODB Connection.open()
    that leads to stale live cache and wrong data provided by database to users.
    
    The bug is that when a connection is opened, it syncs to storage and processes
    invalidations received from the storage in two _separate_ steps, potentially
    leading to situation where invalidations for transactions _past_ opened
    connection's view of the database are included into opened connection's cache
    invalidation. This leads to stale connection cache and old data provided by
    ZODB.Connection when it is reopened next time.
    
    That in turn can lead to loose of Consistency of the database if mix of current
    and old data is used to process a transaction. A classic example would be bank
    accounts A, B and C with A<-B and A<-C transfer transactions. If transaction
    that handles A<-C sees stale data for A when starting its processing, it
    results in A loosing what it should have received from B.
    
    Below is timing diagram on how the bug happens on ZODB5:
    
        Client1 or Thread1                                  Client2 or Thread2
    
        # T1 begins transaction and opens zodb connection
        newTransaction():
            # implementation in Connection.py[1]
            ._storage.sync()
            invalidated = ._storage.poll_invalidations():
                # implementation in MVCCAdapterInstance [2]
    
                # T1 settles on as of which particular database state it will be
                # viewing the database.
                ._storage._start = ._storage._storage.lastTrasaction() + 1:
                    s = ._storage._storage
                    s._lock.acquire()
                    head = s._ltid
                    s._lock.release()
                    return head
                                                            # T2 commits here.
                                                            # Time goes by and storage server sends
                                                            # corresponding invalidation message to T1,
                                                            # which T1 queues in its _storage._invalidations
    
                # T1 retrieves queued invalidations which _includes_
                # invalidation for transaction that T2 just has committed past @head.
                ._storage._lock.acquire()
                    r = _storage._invalidations
                ._storage._lock.release()
                return r
    
            # T1 processes invalidations for [... head] _and_ invalidations for [email protected] transaction.
            # T1 thus will _not_ process invalidations for that next transaction when
            # opening zconn _next_ time. The next opened zconn will thus see _stale_ data.
            ._cache.invalidate(invalidated)
    
    
    The program simulates two clients: one (T2) constantly modifies two integer
    objects preserving invariant that their values stay equal. The other client
    (T1) constantly opens the database and verifies the invariant. T1 forces access
    to one of the object to always go through loading from the database, and this
    way if live cache becomes stale the bug is observed as invariant breakage.
    
    Here is example failure:
    
        $ taskset -c 1,2 ./zopenrace.py
        Exception in thread Thread-1:
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 754, in run
            self.__target(*self.__args, **self.__kwargs)
          File "./zopenrace.py", line 136, in T1
            t1()
          File "./zopenrace.py", line 130, in t1
            raise AssertionError("t1: obj1.i (%d)  !=  obj2.i (%d)" % (i1, i2))
        AssertionError: t1: obj1.i (147)  !=  obj2.i (146)
    
        Traceback (most recent call last):
          File "./zopenrace.py", line 179, in <module>
            main()
          File "./zopenrace.py", line 174, in main
            raise AssertionError('FAIL')
        AssertionError: FAIL
    
    NOTE ZODB4 and ZODB3 do not have this particular open vs invalidation race.
    
    [1] https://github.com/zopefoundation/ZODB/blob/5.5.1-29-g0b3db5aee/src/ZODB/Connection.py#L734-L742
    [2] https://github.com/zopefoundation/ZODB/blob/5.5.1-29-g0b3db5aee/src/ZODB/mvccadapter.py#L130-L139
    """
    
    from __future__ import print_function
    from ZODB import DB
    from ZODB.MappingStorage import MappingStorage
    import transaction
    from persistent import Persistent
    
    # don't depend on pygolang
    # ( but it is more easy and structured with sync.WorkGroup
    #   https://pypi.org/project/pygolang/#concurrency )
    #from golang import sync, context
    import threading
    def go(f, *argv, **kw):
        t = threading.Thread(target=f, args=argv, kwargs=kw)
        t.start()
        return t
    
    
    # PInt is persistent integer.
    class PInt(Persistent):
        def __init__(self, i):
            self.i = i
    
    
    def main():
        zstor = MappingStorage()
        db = DB(zstor)
    
    
        # init initializes the database with two integer objects - obj1/obj2 that are set to 0.
        def init():
            transaction.begin()
            zconn = db.open()
    
            root = zconn.root()
            root['obj1'] = PInt(0)
            root['obj2'] = PInt(0)
    
            transaction.commit()
            zconn.close()
    
    
        okv = [False, False]
    
        # T1 accesses obj1/obj2 in a loop and verifies that obj1.i == obj2.i
        #
        # access to obj1 is organized to always trigger loading from zstor.
        # access to obj2 goes through zconn cache and so verifies whether the cache is not stale.
        def T1(N):
            def t1():
                transaction.begin()
                zconn = db.open()
    
                root = zconn.root()
                obj1 = root['obj1']
                obj2 = root['obj2']
    
                # obj1 - reload it from zstor
                # obj2 - get it from zconn cache
                obj1._p_invalidate()
    
                # both objects must have the same values
                i1 = obj1.i
                i2 = obj2.i
                if i1 != i2:
                    raise AssertionError("T1: obj1.i (%d)  !=  obj2.i (%d)" % (i1, i2))
    
                transaction.abort() # we did not changed anything; also fails with commit
                zconn.close()
    
            for i in range(N):
                #print('T1.%d' % i)
                t1()
            okv[0] = True
    
    
        # T2 changes obj1/obj2 in a loop by doing `objX.i += 1`.
        #
        # Since both objects start from 0, the invariant that `obj1.i == obj2.i` is always preserved.
        def T2(N):
            def t2():
                transaction.begin()
                zconn = db.open()
    
                root = zconn.root()
                obj1 = root['obj1']
                obj2 = root['obj2']
                obj1.i += 1
                obj2.i += 1
                assert obj1.i == obj2.i
    
                transaction.commit()
                zconn.close()
    
            for i in range(N):
                #print('T2.%d' % i)
                t2()
            okv[1] = True
    
    
        # run T1 and T2 concurrently. As of 20191210, due to race condition in
        # Connection.open, it triggers the bug where T1 sees stale obj2 with obj1.i != obj2.i
        init()
    
        N = 1000
        t1 = go(T1, N)
        t2 = go(T2, N)
        t1.join()
        t2.join()
    
        if not all(okv):
            raise AssertionError('FAIL')
        print('OK')
    
    
    if __name__ == '__main__':
        main()
    

    Thanks beforehand, Kirill

    /cc @jimfulton

    P.S. It would be nice to provide ZODB.Connection.at() explicitly similarly to https://godoc.org/lab.nexedi.com/kirr/neo/go/zodb#Connection

  • transaction user and description: text above, bytes below

    transaction user and description: text above, bytes below

    ZODB now translates transaction meta data, user and description from text to bytes before passing them to storages, and converts them back to text when retrieving them from storages in the history, undoLog and undoInfo methods.

    The IDatabase interface was updated to reflect that history, undoLog and undoInfo are available on database objects. (They were always available, but not documented in the interface.)

  • Use a higher pickle protocol for serializing objects on Python 2

    Use a higher pickle protocol for serializing objects on Python 2

    Previously protocol 1 was used. The higher protocol is more efficient for new-style classes (all persistent objects are new-style), according to the docs, at the cost of being very slightly less space efficient for old-style classes.

    In tests of a persistent object with two trivial numeric attributes, the higher protocol was 12 bytes smaller, and serialized and deserialized 1us faster. Introducing a reference to another new-style class (with a small dict and list of strings for attributes) for a more realistic test made the higher protocol twice as fast to serialize (20.5 vs 10.3us), almost half the size (215 vs 142 bytes), and it deserialized 30% faster (6.5 vs 4.6us).

    On Python 2, this will now allow open file objects to be pickled (loading the object will result in a closed file); previously this would result in a TypeError (as does under Python 3). We had tests that you couldn't do that with a BlobFile so I had to update it to still make that true.

    I wouldn't recommend serializing arbitrary open files under Python 2 (for one thing, they can't trivially be deserialized in Python 3), but I didn't take any steps to prevent it either. Since this hasn't been possible, there shouldn't be code in the wild that is trying to do it---and it wouldn't be forward compatible with Python 3 either.

  • KeyError on releasing resources of a Connection when closing the DB.

    KeyError on releasing resources of a Connection when closing the DB.

    UPDATE: see this comment for an up to date description: https://github.com/zopefoundation/ZODB/issues/208#issuecomment-426468941

    When running tests (in plone.restapi) we sometimes get the following KeyError. It is related to timing issues - inserting a time.sleep(2) in the test setup causes the problem to occur more frequently.

    It can also be reproduced outside of a testing environment, when zope is shutting down in the middle of ongoing requests. See a comment below for a method of easy reproduction.

    ZODB 5.4.0 It does not happen with ZODB 5.3.0

    It is related to this commit: https://github.com/zopefoundation/ZODB/commit/1b9475d413705e510ff2f19ee6449688ffb04d46

    In line 948 we see: c.close(False)

    This is part of the traceback: File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 948, in _release_resources c.close(False) In 5.3.0 the corresponding line just sets the transaction manager to None: c.transaction_manager = None

      Set up plone.restapi.testing.PloneRestApiDXLayer:Functional in 0.000 seconds.
      Running:
        1/77 (1.3%) test_comments_add_root (plone.restapi.tests.test_documentation.TestCommenting)/work/buildout_cache/eggs/ZServer-4.0b1-py2.7.egg/ZServer/PubCore/ZServerPublisher.py:22: DeprecationWarning: publish_module is deprecated. Please import from ZServer.ZPublisher.Publish.
      from ZPublisher.Publish import publish_module
        8/77 (10.4%) test_documentation_expansion (plone.restapi.tests.test_documentation.TestCommenting)/work/buildout_cache/eggs/zope.deprecation-4.3.0-py2.7.egg/zope/deprecation/deprecation.py:88: DeprecationWarning: isDefaultPage is deprecated. Import from Products.CMFPlone.defaultpage instead
      name)
        22/77 (28.6%) test_documentation_file (plone.restapi.tests.test_documentation.TestDocumentation)/work/buildout_cache/eggs/Products.PortalTransforms-3.1.3-py2.7.egg/Products/PortalTransforms/libtransforms/commandtransform.py:105: DeprecationWarning: os.popen4 is deprecated.  Use the subprocess module.
      cin, couterr = os.popen4(command, 'b')
        32/77 (41.6%) test_documentation_jwt_login (plone.restapi.tests.test_documentation.TestDocumentation) (1.199 s)Traceback (most recent call last):
      File "./bin/test", line 442, in <module>
        '--test-path', '/work/buildout_cache/eggs/repoze.xmliter-0.6-py2.7.egg',
      File "/work/buildout_cache/eggs/collective.xmltestreport-1.3.4-py2.7.egg/collective/xmltestreport/runner.py", line 60, in run
        failed = run_internal(defaults, args, script_parts=script_parts)
      File "/work/buildout_cache/eggs/collective.xmltestreport-1.3.4-py2.7.egg/collective/xmltestreport/runner.py", line 73, in run_internal
        runner.run()
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 189, in run
        self.run_tests()
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 291, in run_tests
        self.skipped, self.import_errors)
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 460, in run_layer
        import_errors)
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 383, in run_tests
        test(result)
      File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
        return self.run(*args, **kwds)
      File "/usr/lib/python2.7/unittest/case.py", line 370, in run
        result.stopTest(self)
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 897, in stopTest
        self.testTearDown()
      File "/work/buildout_cache/eggs/zope.testrunner-4.8.1-py2.7.egg/zope/testrunner/runner.py", line 821, in testTearDown
        layer.testTearDown()
      File "/work/playground/plone/plone52devel/src/plone.app.testing/plone/app/testing/layers.py", line 288, in testTearDown
        super(PloneTestLifecycle, self).testTearDown()
      File "/work/playground/plone/plone52devel/src/plone.testing/src/plone/testing/z2.py", line 945, in testTearDown
        self['zodbDB'].close()
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 651, in close
        @self._connectionMap
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 518, in _connectionMap
        self.pool.map(f)
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 213, in map
        self.all.map(f)
      File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/weakset.py", line 62, in map
        f(elt)
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/DB.py", line 659, in _
        conn._release_resources()
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 948, in _release_resources
        c.close(False)
      File "/work/buildout_cache/eggs/ZODB-5.4.0-py2.7.egg/ZODB/Connection.py", line 304, in close
        self.transaction_manager.unregisterSynch(self)
      File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/_manager.py", line 106, in unregisterSynch
        self._synchs.remove(synch)
      File "/work/buildout_cache/eggs/transaction-2.2.1-py2.7.egg/transaction/weakset.py", line 51, in remove
        del self.data[id(obj)]
      File "/usr/lib/python2.7/weakref.py", line 86, in __delitem__
        del self.data[key]
    KeyError: 139902031255440
    
    
    
  • Allow serial to be returned as late as tpc_finish

    Allow serial to be returned as late as tpc_finish

    Because of ZODB wrappers like ZEO, I can't touch FileStorage & DemoStorage without breaking compatibility. So what I suggest is to first develop the 2 API in the 4.x branch, with deprecation warnings. Once done, we merge into master branch and (I prefer) force to use the new API.

  • Provide a way to retrieve raw extensions data

    Provide a way to retrieve raw extensions data

    Currently when client has IStorageTransactionMetaData instance (e.g. on storage iteration) it is possible to read transaction's .user and .description in raw form, but .extension is always returned unpickled.

    This creates several problems:

    • tools like zodb dump [1] cannot dump data exactly as stored on a storage. This makes database potentially not bit-to-bit identical to its original after restoring from such dump.

    • zodb dump output could be changing from run to run on the same database. This comes from the fact that e.g. python dictionaries are unordered and so when pickling a dict back to bytes the result could be not the same as original.

      ( this problem can be worked-around partly to work reliably for e.g. dict with str keys - by always emitting items in key sorted order, but it is hard to make it work reliably for arbitrary types )

    Both issues make it hard to verify integrity of database at the lowest possible level after restoration, and make it hard to verify bit-to-bit compatibility with non-python ZODB implementations.

    To fix we provide a way to retrieve raw extension from transaction metadata. This is done in backward-compatible way by introducing new interface

    IStorageTransactionInformationRaw
    

    which can be optionally provided by an IStorageTransactionInformation to indicate that transaction metadata can be also read in raw from.

    Then BaseStorage.TransactionRecord ctor is extended with optional raw_extension argument which if != None is remembered and makes IStorageTransactionInformationRaw to be also provided by constructed TransactionRecord instance.

    We also adapt FileStorage & friends to provide/use new functionality.

    Similarly to e.g. undo, storages should be indicating they will return iterated transactions provided with raw metadata via implementing

    supportsTransactionInformationRaw() -> True
    

    [1] https://lab.nexedi.com/nexedi/zodbtools

  • Modernize race tests and add test that catches ZEO corruption on invalidation / disconnect

    Modernize race tests and add test that catches ZEO corruption on invalidation / disconnect

    Add test for Bug2 described in https://github.com/zopefoundation/ZEO/issues/209 + modernize race testing infrastructure. We add this test here, so that it can be run automatically for all storages - e.g. both ZEO and NEO.

    ZEO currently can corrupt data if client disconnects simultaneously to another client performing tpc_finish. It fails e.g. as follows:

    (z-dev) [email protected]:~/src/wendelin/z/ZEO5$ ZEO_MTACCEPTOR=1 zope-testrunner -fvvvx --test-path=src -t check_race_external_invalidate_vs_disconnect
    /home/kirr/src/wendelin/venv/z-dev/bin/zope-testrunner traceio=True
    /home/kirr/src/wendelin/z/ZEO5/src/ZEO/StorageServer.py:51: DeprecationWarning: The mtacceptor module is deprecated and will be removed in ZEO version 6.
      'in ZEO version 6.', DeprecationWarning)
    Running tests at level 1
    Running .BlobAdaptedFileStorageTests tests:
      Set up .BlobAdaptedFileStorageTests in 0.000 seconds.
      Running:
     check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests) (1.889 s)
    
    Failure in test check_race_external_invalidate_vs_disconnect (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
    Traceback (most recent call last):
      File "/usr/lib/python2.7/unittest/case.py", line 329, in run
        testMethod()
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 357, in check_race_external_invalidate_vs_disconnect
        T2ObjectsInc2Phase())
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/util.py", line 400, in _
        return f(*argv, **kw)
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/racetest.py", line 446, in _check_race_xxx_vs_external_disconnect
        self.fail('\n\n'.join([_ for _ in failure if _]))
      File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
        raise self.failureException(msg)
    AssertionError: T15: obj1 (6) - obj2(4) != phase (1)
    obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
    zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
    zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
    zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor._cache.clear()
    zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: 0x03ea4cc506104155
    zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: 0x03ea4cc506104155
    
    T51: obj1 (6) - obj2(4) != phase (1)
    obj1._p_serial: 0x03ea4cc505486777  obj2._p_serial: 0x03ea4cc503413799  phase._p_serial: 0x03ea4cc505486777
    zconn_at: 0x03ea4cc505486777  # approximated as max(serials)
    zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc503413799  next_serial: None
    zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor._cache.clear()
    zstor.loadBefore(obj1, @zconn.at)       ->  serial: 0x03ea4cc505486777  next_serial: None
    zstor.loadBefore(obj2, @zconn.at)       ->  serial: 0x03ea4cc504a74099  next_serial: None
    zstor.loadBefore(phase, @zconn.at)      ->  serial: 0x03ea4cc505486777  next_serial: None
    

    Please see individual patches for details.

  • with transaction() doesn't work as expected

    with transaction() doesn't work as expected

    I know that there is little development but it might be useful for someone if using ZODB in 2022.

    when trying to store changes to persistent objects inside a ZODB.DB.transaction with block. they are not stored, and no error is raised. while doing the same between transaction.begin() and transaction.commit() calls does work.

    It seems to me that is the only way to currently use a with block is to change objects directly through conn.root(), that means all persistent objects must know full path from root to themselves, which is impractical.

    there is also another weird behavior. where after storing an object for the first time, and retrieving it returns the same object, while the 2nd call and up will return a different object. this trips tests trying to check if something is stored successfully, as it only happens once.

    the following code tries to store attributes in a 2 level persistent hierarchy (represents actual code)

    import ZODB
    import ZODB.FileStorage
    from persistent.mapping import PersistentMapping
    import transaction
    
    store=ZODB.FileStorage.FileStorage("temp1.db")
    db=ZODB.DB(store)
    
    def get_init(name, obj):
        with db.transaction(f"creating root[{name}]") as conn:
            try:
                return conn.root()[name]
            except KeyError:
                conn.root()[name] = obj()
                return conn.root()[name]
    
    class A:
        def __init__(self):
            self.cfg = PersistentMapping()
    
        def __setitem__(self, key, value) -> None:
            transaction.begin()
            self.cfg[key+", inside block"] = value
            transaction.commit()
            with db.transaction():
                self.cfg[key+", inside with"] = value #does not work
            #these should be equivalent, no?
    
        def __iter__(self):
            return iter(self.cfg)
    
    class Manager:
        def __init__(self):
            self.a1=get_init("testing", PersistentMapping) # set up the db, should only happen once
    
        def __setitem__(self, name, obj) -> None:
            """Registers in persistent storage"""
            with db.transaction(f"Adding testing:{name}") as conn:
                if name in conn.root()["testing"]:
                    print(f"testing with same name {name} already exists in storage")
                    return
                conn.root()["testing"][name] = obj
    
        def __getitem__(self, name: str):
            return db.open().root()["testing"][name]
    
    dm=Manager()
    
    initial=A() #only relevant for forst run
    dm['a']=initial #only relevant for forst run
    
    fromdb1= dm['a']
    fromdb2= dm['a']
    
    with db.transaction() as conn:
        fromdb1.cfg['updated from outer txn, directly'] = 1 #doed not work
        conn.root()['testing']['a'].cfg['updated from outer txn,through conn'] = 1
        #this should be equivalent but only the second one works
    
    # transaction.begin()
    # transaction.commit()
    initial['new txn updated on initial'] = 1
    fromdb1['new txn updated on retrieved 1']= 1
    fromdb2['new txn updated on retrieved 2']= 1
    # db.open().root()['testing']['a']['new txn updated on new db']=1
    
    print(f"initial obj - {initial.cfg}")
    print(f"from db obj 1- {fromdb1.cfg}")
    print(f"from db obj 2- {fromdb2.cfg}")
    print(f"\nnew from db obj- {dm['a'].cfg}")
    
    print(f"\nis the initial obj and the first obj from db the same: {initial is fromdb1}")
    print(f"is the initial obj and the second obj from db the same: {initial is fromdb2}")
    

    The expected result is for all writing methods to work similarly, or the docs to reflect those limitations.

    Thanks!!

    What version of Python and Zope/Addons I am using:

    Python 3.9.2, ZODB3, Win1064bit

  • FileStorage and other api doesn't support pathlib interface.

    FileStorage and other api doesn't support pathlib interface.

    BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

    For modern python library which involved path location, it would be nice to accept Union[str, os.PathLike[str]], instead of just str.

  • Dual Inquiry: 2022 Best Practices for ZODB usage outside of ZOPE using ZEO SharedStorage

    Dual Inquiry: 2022 Best Practices for ZODB usage outside of ZOPE using ZEO SharedStorage

    I have a bit of a dual question: 1st. Is there a better place I should be asking these non-issue clarification questions that the docs alone were unable to answer for me? 2nd: What are a some best practices for using ZODB in a ZEO configuration with a handful of users that will be operating(r/w) mostly in their own BTree persistences? specifically I am trying to employee best practices when trying to read from the database such that I don't overwrite work by attempting to refresh the current view of the 'live' data.

    Any assistance would be much appreciated as I am very new to ZODB and have chosen it as we need a near embedded database instance that can be easily traversed to find groupings of applicable data where the structure can be vastly different between the entries beyond core 'queryable' tags that are used in the keys of the BTree structures mostly.

  • 5.7.0: pytest warning and ZODB/tests should not be installed

    5.7.0: pytest warning and ZODB/tests should not be installed

    I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

    • python3 -sBm build -w --no-isolation
    • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
    • install .whl file in </install/prefix>
    • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

    Here is pytest output:

    + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zodb-5.7.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zodb-5.7.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
    + /usr/bin/pytest -ra
    ==================================================================== test session starts =====================================================================
    platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/ZODB-5.7.0
    collected 96 items
    
    src/ZODB/scripts/tests/test_doc.py .                                                                                                                   [  1%]
    src/ZODB/scripts/tests/test_fsdump_fsstats.py ..                                                                                                       [  3%]
    src/ZODB/scripts/tests/test_fstest.py ..                                                                                                               [  5%]
    src/ZODB/scripts/tests/test_repozo.py ........................................................................                                         [ 80%]
    src/ZODB/tests/testConnectionSavepoint.txt .                                                                                                           [ 81%]
    src/ZODB/tests/test_TransactionMetaData.py .......                                                                                                     [ 88%]
    src/ZODB/tests/test_cache.py .                                                                                                                         [ 89%]
    src/ZODB/tests/test_doctest_files.py .                                                                                                                 [ 90%]
    src/ZODB/tests/test_fsdump.py .                                                                                                                        [ 91%]
    src/ZODB/tests/test_mvccadapter.py ...                                                                                                                 [ 94%]
    src/ZODB/tests/test_prefetch.py ....                                                                                                                   [ 98%]
    src/ZODB/tests/test_storage.py .                                                                                                                       [100%]
    
    ====================================================================== warnings summary ======================================================================
    src/ZODB/scripts/tests/test_fsdump_fsstats.py::FsdumpFsstatsTests::test_fsstats
      /usr/lib64/python3.8/runpy.py:127: RuntimeWarning: 'ZODB.scripts.fsstats' found in sys.modules after import of package 'ZODB.scripts', but prior to execution of 'ZODB.scripts.fsstats'; this may result in unpredictable behaviour
        warn(RuntimeWarning(msg))
    
    -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
    =============================================================== 96 passed, 1 warning in 1.19s ================================================================
    

    Also looks like ZODB/test/ content is installed. I think that this warning may be related to fact that ZODB/tests is installed.

  • Blob usage depending on file size

    Blob usage depending on file size

    Is there a best practice or some first hand experience for which file sizes it makes sense to use Blob storage?

    I'm using the blob storage to store files and images, but I wonder if it's maybe better (performance-wise?) to keep small files in the ZODB.

  • Change `pack_date` interpretation?

    Change `pack_date` interpretation?

    Currently, FileStorage.pack deletes non current states when they have been created before pack_date (the parameter is aktually named t, a timestamp).

    While this leads to an easy implementation, it does not conform to the intuitive interpretation: I have the complete history up to pack_date (and can revert up to this date). For this, we would need one additional older state.

    With the current interpretation, even a very old pack_date does not guarantee that a very recent modification (to a very stable object) can be reverted. Should we change the pack_date itnerpretation to include one additional older state?

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database.

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database. This database is completely free* ??

Nov 25, 2022
Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Nov 9, 2022
LightDB is a lightweight JSON Database for Python

LightDB What is this? LightDB is a lightweight JSON Database for Python that allows you to quickly and easily write data to a file Installing pip3 ins

Oct 1, 2022
A Simple , ☁️ Lightweight , 💪 Efficent JSON based database for 🐍 Python.
A Simple , ☁️ Lightweight ,  💪 Efficent JSON based database for 🐍 Python.

A Simple, Lightweight, Efficent JSON based DataBase for Python The current stable version is v1.6.1 pip install pysondb==1.6.1 Support the project her

Nov 27, 2022
A Persistent Embedded Graph Database for Python
A Persistent Embedded Graph Database for Python

Cog - Embedded Graph Database for Python cogdb.io New release: 2.0.5! Installing Cog pip install cogdb Cog is a persistent embedded graph database im

Nov 21, 2022
A Painless Simple Way To Create Schema and Do Database Operations Quickly In Python
A Painless Simple Way To Create Schema and Do Database Operations Quickly In Python

PainlessDB - Taking Your Pain away to the moon ?? Contribute · Community · Documentation ?? Introduction : PainlessDB is a Python-based free and open-

Jul 15, 2022
HTTP graph database built in Python 3

KiwiDB HTTP graph database built in Python 3. Reference Format References are strings in the format: {[email protected]} Authentication Currently, t

Dec 17, 2021
A NoSQL database made in python.

CookieDB A NoSQL database made in python.

Dec 23, 2021
Tiny local JSON database for Python.
Tiny local JSON database for Python.

Pylowdb Simple to use local JSON database ?? # This is pure python, not specific to pylowdb ;) db.data['posts'] = ({ 'id': 1, 'title': 'pylowdb is awe

Jan 26, 2022
Shelf DB is a tiny document database for Python to stores documents or JSON-like data
Shelf DB is a tiny document database for Python to stores documents or JSON-like data

Shelf DB Introduction Shelf DB is a tiny document database for Python to stores documents or JSON-like data. Get it $ pip install shelfdb shelfquery S

Nov 3, 2022
This is a simple graph database in SQLite, inspired by
This is a simple graph database in SQLite, inspired by

This is a simple graph database in SQLite, inspired by "SQLite as a document database".

Nov 30, 2022
Elara DB is an easy to use, lightweight NoSQL database that can also be used as a fast in-memory cache.
Elara DB is an easy to use, lightweight NoSQL database that can also be used as a fast in-memory cache.

Elara DB is an easy to use, lightweight NoSQL database written for python that can also be used as a fast in-memory cache for JSON-serializable data. Includes various methods and features to manipulate data structures in-memory, protect database files and export data.

Nov 9, 2022
A simple GUI that interacts with a database to keep track of a collection of US coins.

CoinCollectorGUI A simple gui designed to interact with a database. The goal of the database is to make keeping track of collected coins simple. The G

Nov 9, 2021
Makes google's political ad database actually useful

Making Google's political ad transparency library suck less This is a series of scripts that takes Google's political ad transparency data and makes t

Apr 28, 2022
MyReplitDB - the most simplistic and easiest wrapper to use for replit's database system.

MyReplitDB is the most simplistic and easiest wrapper to use for replit's database system. Installing You can install it from the PyPI Or y

Jul 3, 2022
Decentralised graph database management system

Decentralised graph database management system To get started clone the repo, and run the command below. python3 database.py Now, create a new termina

Apr 18, 2022
PathfinderMonsterDatabase - A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

PathfinderMonsterDatabase A database of all monsters in Pathfinder 1e, created by parsing aonprd.com Setup Run the following line to install all requi

Jun 12, 2022
ClutterDB - Extremely simple JSON database made for infrequent changes which behaves like a dict

extremely simple JSON database made for infrequent changes which behaves like a dict this was made for ClutterBot

Jan 12, 2022
A very simple document database

DockieDb A simple in-memory document database. Installation Build the Wheel Fork or clone this repository and run python setup.py bdist_wheel in the r

Jan 16, 2022